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Abstract 


Diagnostic tests have been developed previously to measure algebraic thinking skills; however, the tests 
do not specifically address algebraic problem-solving. Thus, an Algebraic Thinking Test (ATT) Instrument 
was developed to measure algebraic thinking skills in problem-solving involving linear equations. ATT 
comprises nine open-ended questions with three algebraic thinking constructs: Generalized Arithmetic, 
Functions and Modelling. Generalized arithmetic involves students in efficient calculation and 
generalization; functional involves identifying number patterns, while modelling involves solving open- 
ended problems, identifying similarities, and performing calculations involving variables. This study is 
meant to determine the quality of ATT instruments through the validity and reliability analysis using 
the Rasch Measurement Model. The sample consisted of 120 seventh graders aged 12 to 13, selected 
from two secondary schools in the Tuaran district, Malaysia. The instrument was found to have a strong 
dimensionality and high construct validity. The reliability of Cronbach Alpha (KR-20) demonstrated a 
value of .90 (very high), and item and respondent reliability of .98 (excellent) and .86 (good), respectively, 
with an item separation index of 6.29 and 2.45 for the person separation index. ATT has good validity 
and high reliability in measuring algebraic thinking skills among seventh graders in secondary schools. 

Keywords: algebraic thinking skills, linear equations, problem-solving, Rasch analysis, validity and 
reliability 


Introduction 


Thinking skill is a distinct human capability. Nickerson (1985) defines thinking skills 
as a cognitive process an individual uses to solve problems. Beyer (1991) further elaborates 
that thinking skills are rooted in human abilities to form concepts, reason, and make decisions. 
Algebraic thinking skills are crucial components of mathematics education and have been the 
foundation of the subject for centuries (Carraher et al., 2007). Ibrahim and Othman (2010) 
suggested that algebra should be introduced to students at an early stage to stimulate their 
thinking abilities. For example, scholars such as Carraher and Schliemann (2007), Katz (2007), 
and National Council of Teachers of Mathematics (NCTM) (2000) argued that algebra, which 
involves reasoning, should be integrated into primary school curricular as a preparation for 
students to learn algebra in secondary school. 

Mathematical processes are considered crucial for teaching and learning mathematics. 
These processes provide the groundwork for developing problem-solving abilities, which can 
be further enhanced through the application of various problem-solving strategies. As students 
learn to employ these strategies, they can develop critical thinking, creativity, and innovation 
skills. 

In the past, the learning of algebra was mainly centred on symbol manipulation and 
arithmetic procedures, without much connection to the real world. According to Van Amerom 
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(2002), this led students to perceive algebra as an abstract concept, making it challenging for 
them to understand. Additionally, the teacher focused on rote memorization of formulas to obtain 
correct exam answers without a deep understanding of the concepts and reasons. However, the 
current approach has shifted to training students in high-level thinking skills (HOTS) questions, 
which promote the development of conceptual, problem-solving, and application skills (Smith, 
2014) related to algebraic thinking skills. 


Research Problem 


The average performance of Malaysian students in lower secondary schools in mastering 
algebraic concepts is still below par, as indicated by the results of the Trends in International 
Mathematics and Science Study (Mullis, 2019). Malaysia is ranked 29th out of 42 countries, 
and only a small percentage of students (5%) were able to apply algebraic problem-solving 
skills involving application, reasoning, and generalization. Similarly, studies in Finland, Swe- 
den, and South Africa found that students struggled to master algebraic concepts, particularly 
equality and function. According to research by Bishop and Stump (2000), Jacobs et al. (2007) 
and Stephens (2008), one of the reasons for this difficulty is a lack of understanding of algebraic 
features. This deficiency is exacerbated by the fact that algebra is not formally introduced to 
students at the primary school level, which is emphasized more on arithmetic skills as reflected 
in Malaysian's Standard Primary School Curriculum. 

In 2019, there has been a notable rise in emphasis on the development of algebraic 
thinking abilities in learning (Sibgatullin et al., 2022). Despite the attention given to this topic 
by researchers, there is a lack of instruments available to measure the level of algebraic thinking 
skills aligned with the Malaysian curriculum. The scarcity of assessment tools available to 
teachers for evaluating their students' algebraic thinking skills has contributed to this problem. 
Although a diagnostic test has been developed previously to measure generalization, function, 
and modelling skills (Ralston, 2013), it does not specifically address algebraic problem-solving. 
Consequently, the researcher developed a new instrument and assessed its validity and reliability 
to ensure its applicability in field research, hence to provide a reference for other researchers in 
the field to study the algebraic thinking abilities of secondary school students. 


Literature Review 
Algebraic Thinking Skills 


Algebra is a branch of mathematics that employs symbols and letters as representations 
of values and quantities, and it plays a crucial role in other mathematical disciplines such as 
arithmetic, geometry, statistics, and calculus (Sianturi & Yang, 2021; Saundarajan et al., 2020). 
However, when students are formally introduced to algebraic topics at a lower secondary level, 
they often feel disinterested and not concerned (Lim & Lau, 2015). This leads to a barrier to 
developing algebraic thinking skills. Algebraic thinking skills refer to a process in learning 
mathematics that helps students to generalize mathematical ideas from examples to writing and 
arguments and statements (Hadi & Faradillah, 2019). Kaput et al. (2008) classified algebraic 
thinking skills into three main components: generalized arithmetic, function, and modelling. 
Ralston (2013) has further elaborated on the related dimensions of these three constructs. The 
construct of generalized arithmetic involves efficient calculation and generalization. The con- 
struct of functional thinking skills involves identifying number patterns, while modelling in- 
volves solving open problems, identifying similarities, and performing calculations involving 
variables. 
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Generalized Arithmetic 


According to Kaput et al. (2008), generalized arithmetic is the study of algebra involving 
the examination of structures and systems that relate to computational and relational laws with- 
in mathematics. Fujii and Stephens (2001) suggested that students should be exposed to gener- 
alized arithmetic as an evolution from arithmetic to algebraic thinking skills. The development 
of algebraic thinking is described in some literature as a process that involves identifying and 
representing structures in mathematical expressions (Mason, 1989; Sfard & Linchevski, 1994). 
This process is related to an individual's cognitive ability to extract and establish generaliza- 
tions, and to use appropriate forms of representation to express those generalizations (Carraher 
& Schliemann, 2007). This ability is the foundation for developing algebraic thinking skills, 
beginning with the basics of arithmetic (Carpenter et al., 2003; Jacobs et al., 2007). 


Function 


In Blanton and Kaput (2005)'s research, it was discovered that even primary school 
children have the ability to use different representations concerning function reasoning. This 
includes their capacity to verbally and symbolically describe recursive relations, covariations, 
and data matching and their ability to model and solve equations with unknown quantities 
using symbolic language. Blanton and Kaput's findings are supported by studies conducted by 
Brizuela and Lara-Roth (2000), Carraher et al. (2007), Moss and London McNab (2011), and 
Carraher and Schliemann (2007). In Lins and Kaput (2004)’s study with fifth-grade students, it 
was found that functional algebraic thinking skills can be developed through scaffolding. The 
gradual introduction of tables, graphs, pictures, words and symbols can shape students' thinking 
so that they can understand data and interpret functional relations more effectively. 


Modelling 


According to Ralston (2013), modelling is an essential component of algebraic thinking 
skills and involves solving open-number sentences, understanding similarities, and performing 
calculations involving variables. The use of variables and unknown values is indirectly 
introduced to students through activities that require them to find a value in an empty space, 
allowing them to understand the relations between arithmetic operations. Many researchers also 
emphasize equality as a means to develop algebraic thinking skills among students (Carpenter 
et al., 2005; Rittle-Johnson & Alibali, 1999). The symbol "equal to" has three distinct meanings: 
indicating that two quantities are equal, representing a relation, and suggesting the existence 
of two sides of an equation (Rittle-Johnson & Alibali, 1999, p.177). However, studies have 
shown that many students struggle with the conceptual understanding of variables, especially 
regarding the symbolic use of variables in algebra (Ktichemann, 1978; Usiskin, 1988). 


Algebraic Problem Solving 


Solving algebraic problems is important because it helps students understand real- 
world situations and build meaning in real life (Kim & Chang, 2010; NCTM, 2000). However, 
students may require guidance in solving algebraic problems (Capraro & Joffrion, 2006), as 
these problems tend to be less popular among students due to their abstract nature (Baysal 
& Sevinc, 2021). Consequently, students often struggle with the understanding of problem 
statements, identifying variables, expressing known and unknown relations, comprehending 
information, and translating algebraic problems into expressions or equations, leading to a 
tendency to make mistakes (Clements, 1980; Egodawatte, 2011; Nathan & Koedinger, 2000). 
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Ralston et al. (2018) developed a diagnostic assessment tool with 25 items, containing 
eight components of algebraic skills related to 1745 elementary school students, demonstrating 
good item fit but low reliability. There is still a need for a comprehensive instrument that 
incorporates the constructs of generalized arithmetic, function, and modelling through problem- 
solving. To fill this gap, this study developed and tested an assessment tool for its validity and 
reliability that focuses on measuring these three constructs of algebraic thinking skills among 
high school students in terms of problem-solving ability. 


Rasch Measurement Model 


The Rasch Measurement Model (RMM) has been employed to measure validity and 
reliability in ATT items. This model is a useful tool for obtaining high instrument validity 
and reliability levels through advanced statistical techniques (Bond & Fox, 2007). The RMM 
assesses respondents' abilities to answer the instrument and evaluates the difficulty level of 
each item (Wolins, 1982). Additionally, the RMM can measure underlying characteristics such 
as human thoughts and emotions (Azrilah et al., 2015). While Rasch analysis may require a 
longer process than traditional analysis, it offers a more in-depth understanding of the strengths 
and weaknesses of the instrument (Boone & Scantlebury, 2006). According to Bond and Fox 
(2007), the RMM effectively creates highly valid and reliable instruments through statistical 
analysis. 

The Rasch Measurement Model (RMM) is based on Item Response Theory and is 
considered an appropriate statistical model as it can simultaneously measure the difficulty and 
ability of individuals to respond to the items (Deane et al., 2016). The RMM analysis provides 
information about the reliability of individuals and items, the separation of item and person 
indices, and Cronbach's alpha value. Moreover, the construct validity of an instrument can be 
evaluated through item and person fit, as well as unidimensionality. The researcher used Rasch 
analysis to establish the reliability and validity of the ATT instrument by applying these key 
concepts. 


Research Aim and Questions 
The aim of this study was to develop a reliable and valid tool for assessing algebraic- 
related problem-solving abilities among seventh-grade students, using the Rasch Measurement 


Model. The study was guided by three research questions: 


QI: Is ATT feasible in assessing algebraic thinking skills related to problem-solving 
among seventh graders? 


Q2: How valid is ATT in assessing seventh graders’ algebraic thinking skills related to 
problem-solving? 


Q3: How reliable is ATT in assessing the algebraic thinking skills involving problem- 
solving among seventh graders? 
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Research Design 


This research utilized a descriptive research design. The descriptive research design 
focused on evaluating the validity and reliability of the ATT instrument. The development of 
ATT involved a four-step process, which included construct identification, item formulation, 
pre-testing ATT, and validity and reliability testing. Prior to item construction, a literature review 
was conducted to gain an overview of the construct of algebraic thinking and the objectives 
of the national curriculum. This was followed by a comparison of the learning objectives of 
Mathematics in problem solving, as well as the results of TIMSS and PISA in algebra. Item 
pre-testing took place from 10 to 24 August 2022 to check if the questions and instructions were 
clear, after which modifications and refinements were made. The validity and reliability testing 
was conducted on 5 October 2022. 


Sample 


The research employed a purposive sampling technique. The selection of the research 
sample was based on the criteria that the students possessed a similar level of mathematics 
achievement as assessed by their teachers in the end-of-semester test marks (70-80%). The 
schools were categorized in the middle band in terms of mathematics achievement based on 
the Subject Average Grade set by the Tuaran District Education. A total of 120 seventh graders, 
consisting of 64 females (53.3%) and 56 males (46.7%) aged 12 to 13, were randomly selected 
from two rural secondary schools in the Tuaran district of Sabah, Malaysia. According to 
Linacre (1994), a sample size ranging from 108 to 243 can provide reliable and consistent data 
with a 99% confidence level for estimating item difficulty within + 4 logit of its stable value. 


Instrumentation 


The Algebraic Thinking Test (ATT) consists of nine open-ended subjective questions. 
Multiple approaches can be taken to solve these subjective questions. As noted by Tofade et al., 
(2013), the process of answering these types of subjective questions is more vital to learning 
than attaining the correct answers as seen in objective questions. As such, these subjective 
questions promote high-level thinking skills. Table 1 gives more information regarding the 
ATT. 


Table 1 
ATT Format 
Format Description 
Skills tested Algebraic problem solving based on the construct of Algebraic Thinking 
Duration 1 hour 15 minutes 
Number of Items 9 
Language Use Malay / English 
Total Marks 45 
Total Full Score (%) 100 
Type of Item Subjective (Open-ended response) 
Applying 
Cognitive level Analyze 
Evaluate 
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The ATT items were developed based on the content of the Curriculum Standard of 
Secondary School (CSSS) as well as the Standard Document of Curriculum and Assessment 
(SDCA) of Form One Mathematics under the field of Relations and Algebra (Curriculum 
Development Division, 2017: 53 — 56). Based on SDCA, with discussion and advice from the 
Coach of CSSS Mathematics and a Head of Mathematics, the researcher built a test specification 
table (TST) for one of the topics involved, as shown in Table 2. 


Table 2 
Algebraic Thinking Skills Test Specification Table 


Item Content Standards Learning Standards Performance Standard Difficulty Level 


61 62 63 612 61.3 622 632 GA F M Low Medium — High 
/ / / / 


/ 
/ 
/ 
/ 
/ 
/ 
/ 
/ 
/ 


O};/O; NID) a] eI] w]rm]— 


Total 4 3 2 


Each question in ATT includes three algebraic thinking skill constructs adapted from 
Kaput et al. (2008)’s algebraic thinking skills model: generalized arithmetic (GA), function (F), 
and modelling (M). Table 3 provides examples of questions based on each construct, while the 
solutions are illustrated in Figure 1. Each ATT item has a scoring system ranging from 0 to 6, 
with a minimum and maximum score of 0 and 6 respectively. The student's response determines 
the score awarded for each item, with a range of 0 to 6 marks for each item. 


Table 3 
The Constructs of ATT and Sample Questions 


Sample Questions in ATT: The sum of three odd numbers in a row is 237. What is the number? 


Constructs Definition Ability tested in the sample question 
‘ 1. Using arithmetic laws and being able to Students can show an understanding that 
Generalized : mis 
: ; manipulate numbers efficiently. odd numbers have a sequence plus two 
Arithmetic (GA) : ; ; : ; 
2. Doing generalized arithmetic. from the previous odd number. 


1. Specifying the relations between two or more 


variables in the form of quantity. Students can express the sequence of odd 


runes) 2. Expressing the relations between two or more numbers lita peas involving 
variables as a number pattern. ‘ 
1. Using variables, especially in problem-solving. : : 
Modelling (M) 2. Understanding the "equal" symbol means SelM ete sinas Ia elution 


similarity. 
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Figure 1 229 
Demonstration of a Solution Using Conceptual Components of Algebraic Thinking 


Pattern of odd numbers x , x + 2, GA 


Pattern of three consecutive odd numbers x,x + 2,x+2+4+2 


Sum of three consecutive odd numbers 
x+xX4+2+ X+242 = 237 
3x + 6 = 237 
3x + 6—6 = 237 —6 


3x 


x= 


Three consecutive odd numbers, 77,79,81 


Figure 2 shows an example of the problem posed to the students in ATT. Attempting 
to solve the problem, the student was able to demonstrate the ability to apply the concept of 
variables in equality. The student also attempted to indicate the relation between variable and 
equality by showing that the sum of 7x, 10, and 30 are the same as 180. To solve the problem, 
the student manipulated the equality concept and the numbers effectively by subtracting 40 
from both sides of the equation, hence obtaining the value of x. 
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Student's Answer for Item 7 


Gambarajah 1 menunjukkan sebuah segi tiga POR. Cari nilai x. 
(_ hasil tambah sudut pendalaman = 180°]. 


Diagram 7 shows a triangle POR. Find the value of 2 
{ the sum of interior angles = ¥s0e 


Gambarajah / Diagram / 


Jawapan / Answer; 


; Variable and equality 
[304 105) 4 C2450") 4 FUND * 180 concept 


Using arithmetic laws and 
manipulate the numbers. 


Meanwhile, in Figure 3, the student not only was able to demonstrate the ability to use 
variables and equality but also was able to specify the relation between the two variables. 
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Figure 3 
Student's Answer for Item 8 


ory. = = her 
In a group of 67students, there are\9 boys morejkan girls students. Find the num 
of boyy students in the group. 


Jawapan / Auswer: 


Specify the relation between 
the two variables in the form 
of quantity 


Variable and equality concept 


Using arithmetic laws and 
manipulate the numbers 


number 6f bey 
Students 


Ethical Considerations 


Prior to the administering of ATT, written consent was acquired from the participants’ 
parents and school principals. The consent letter contained information about the students’ in- 
volvement in the research, and the parents indicated that they had a clear understanding of the 
research objectives. Other than that, they were also provided with an outline of ATT’s purpose 
to measure the level of algebraic thinking skills of the students. Participants were reassured that 
their responses would be kept confidential and anonymous, and that they were free to pull out 
of the study at any time without any consequences. Afterwards, a follow-up briefing was con- 
ducted to explain the instructions regarding how to answer the questions in ATT. 


Administration of ATT 


During the assessment, the students were instructed to read the ATT attentively, use their 
own knowledge to answer the questions, and avoid seeking assistance from external sources. 
They were given one hour to analyze and respond to the open-ended prompts. Following the 
completion of the task, the responses were collected, verified for correctness, and analyzed 
utilizing WINSTEPS software version 3.73. 
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Content Validity 


Validity pertains to the precision of the measurement in an instrument. Content validity 
evaluates the adequacy of the items in representing the trait's content that the researcher intends 
to measure, and experts' review ensures construct precision and content clarity, as cited by 
Creswell & Creswell (2017) and Kline (2005). To assess content validity value, the researcher 
utilized the Content Validity Index (CVI), which assesses the rating for all items evaluated by 
experts. Previous research specified that the Scale Content Validity Index (S-CVIJ) of .80 or 
higher (Davis, 1992) when the assessment of content validation involves three or more experts 
for a new instrument. This study applied a benchmark of .80, proposed by Davis (1992) for 
ATT. The analysis of CVI, based on Polit and Beck (2006), is presented below. 


Content Validity Index (CVI) = (Total score agreed by experts) 
(Total Score) 


To enhance ATT's validity, the research conducted a content validity process with the 
assistance of four Mathematical Education experts. The researcher adopted an evaluation form 
suggested by the Malaysian Examinations Board to evaluate ATT items in terms of the suitability 
of algebraic thinking skills, language clarity, and item relevance. The experts' feedback was 
documented and used to improve the ATT items and refine their meaning, language, and 
presentation. Table 4 provides a list of the experts who participated in content validation. 


Table 4 
Content Validation Panel for ATT 


Expert Representative Position Expertise 
Expert A Buble Uawersiies Associate Professor Measurement and Assessment in 
Mathematical Education 
Expert B Public Universities Academic Lecturer (PhD) Mathematics Education, Statistics 
Expert C Institute of Teacher Education Lecturer (PhD) Mathematical Education 
Lecturer 
Head Coach of KSSM HOTS in Mathematics, Mathematics 
Expert D tnetifute:of Teacher Education Mathematics, Curriculum, Digital Mathematical 
Head Coach of Literacy and Mathematical 
Mathematics Educator, Assessment 


Mathematics’ Examiner 


Table 5 shows that the Scale content validity index yielded a value of .94, surpassing the 
minimum requirement of .80, as Davis (1992) stipulated for new instruments. A S-CVI score of 
0.94 indicates that the ATT instrument has a very high level of content validity, as assessed by 
the expert panel, and is deemed acceptable. 
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Table 5: 
Content Validity Index (CVI) Result of the ATT instrument 


Item = Reeentt I-CVI Decision 

1 2 3 4 Expert 
1 / / / / 4 1.00 Maintained 
2 / / / 3 15 Modified 
3 / / / 3 15 Modified 
4 / / / / 4 1.00 Maintained 
5 / / / / 4 1.00 Maintained 
6 / / / / 4 1.00 Maintained 
7 / / / / 4 1.00 Maintained 
8 / / / / 4 1.00 Maintained 
9 / / / / 4 1.00 Maintained 
Scale Content Validity Index (S-CVI/ Average) .94 


Construct Validity 


The application of five statistical tests is required to conduct a basic Rasch analysis, 
which involves: (1) Measuring item polarity through Point Measure Correlation (2) Measuring 
item fit through the mean square (3) Confirming dimensionality (4) Determining Cronbach's 
alpha coefficient (KR-20) (5) Determining both item and person reliability by assessing their 
respective separation indices. 

The item polarity is determined by the Point Measure Correlation (PTMEA-CORR) 
value analysis. A positive PTMEA-CORR value indicates that the item accurately measures its 
intended measure, and a negative value indicates the opposite. Additionally, item fit analysis 
was conducted using Outfit MNSQ, Outfit ZSTD, and PTIMEA-CORR values, as suggested 
by Bond and Fox (2007), Boone et al. (2014), and Leitao and Waugh (2012). The purpose of 
item fit analysis is to determine if an item can perform its intended measurement (Sumintono 
& Widhiarso, 2015). Any items that fail to meet the criteria in Table 6 should be revised or 
removed to increase the item's fair value (Sumintono & Widhiarso, 2015). 

To ensure the instrument's ability to measure algebraic thinking proficiency, the 
researcher conducted a dimensionality analysis using Principal Component Analysis (PCA), 
in accordance with Shea et al. (2009) and Sumintono and Widhiarso (2015). The dimensional 
criteria were based on the "raw variance explained by measures" value, which is considered 
good if it exceeds 20%, excellent if it exceeds 60%, and 40% or more is considered acceptable 
(Sumintono & Widhiarso, 2015). Additionally, the "unexplained variance in first contrast" 
value should not exceed 15%. 
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Table 6 
Fit Indices for Item Fit 
Statistics Fit Indices 
Outfit Means Square Values (MNSQ) 50 - 1.50 
Outfits Z-Standard Values (ZSTD) 2.00 - 2.00 
AQ - .85 


Point Measure Correlation (PTMEA CORR) 


Source: Boone et al. (2014) 
Reliability 


Sumintono and Widhiarso's (2015) work is referenced by the researcher for evaluating 
reliability in terms of Cronbach's Alpha (KR-20) values, as well as item-person reliability and 
separation indices, as shown in Table 7. 


Table 7 
Reliability Measured via Rasch Analysis 


Statistics Index Interpretation 
<5 Low 
<.6 Moderate 

Alpha Cronbach (KR-20) 6-7 Good 

1-8 High 
9-1.0 Very High 
<.67 Low 
.67 — .80 Sufficient 
Item and Person Reliability Index 81-90 Good 
.91- .94 Very Good 
>.94 Excellent 


Ahigh exile value indicates good persistence 
between the item and the person and describes 
the strata of the item and person in the 
instruments. 


Item and Person Separation Index > 3.0 


Source: Sumintono and Widhiarso (2015) 
Research Results 
Item Statistics 
Linacre (2007) emphasized the importance of Outfit MNSQ values in evaluating the 
suitability of measurement items for a construct. For productive items, Boone et al. (2014) 
specified that the Outfit MNSQ range should be between .5 and 1.5, the Outfit ZSTD range 


should be within -2.0 to +2.0, and the PTMEA-CORR range should fall between .4 to .85. 
Based on Table 8, item 5's Outfit MNSQ value falls outside the range. As explained by Boone 
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et al. (2014) and Abdul Aziz et al. (2014), an item that does not meet all three criteria and falls 
outside the range is considered unsuitable. However, if an item meets any criteria, it should be 
retained, as suggested by Sumintono and Widhiarso (2015). Since all items meet at least one 


criterion in Table 8, no changes were made to the instrument. 


Table 8 
Item Statistics 


Outfit MNSQ 


Outfit ZSTD 


PTMEA-CORR 


iin (.50 - 1.50) (-2.0 - 2.0) (.40 - 85) Dycision 
5 1.94 5.10 A6 Remain 
1 A8 - 4.70 92 Remain 
6 82 - 1.30 Ha Remain 
7 93 - 40 81 Remain 
2 1.10 60 68 Remain 
3 91 -.40 67 Remain 
4 86 - 50 66 Remain 
8 92 - 20 68 Remain 
9 A2 - 2.10 AD Remain 
Item Polarity 


The assessment of item polarity was conducted using the PTMEA-CORR values, which 
determined whether the items in ATT moved congruently with the measured construct. A 
positive value indicated that all items functioned consistently, while a negative value indicated 
items that required improvement or elimination. As shown in Figure 4, all items had positive 
values, with PTMEA-CORR scores ranging from .46 to .92. 


Figure 4 


Analysis of Point Measure Correlation 


INPUT: 169 Person 9 Item REPORTED: 109 Person 9 Item 4 CATS 


Person: REAL SEP.: 


Item STATISTICS: 


[NUMBER SCORE 


| MEAN 439.1 


WINSTEPS 3.73 


2.45 REL.: .86 ... Item: REAL SEP.: 6.29 REL.: .98 
MEASURE ORDER 

TOTAL MODEL] INFIT | OUTFIT |PT-MEASURE |EXACT MATCH| l 
COUNT MEASURE S.E. |MNSQ ZSTD|MNSQ ZSTD|CORR. EXP.| OBSX EXPX| Item | 
|------------------------------------ +---------- $---------- +--2-------- +----------- +------ l 
109 71.41 1.75]1.85 4.9]1.94 5.1] |.46] .74] 43.1 66.3| 10005] 
109 68.05 1.71] .51 -4.6| .48 -4.7] |.92| .74] 77.1 61.3] 10001] 
109 558.60 1.71] .82 -1.4] .82 -1.3] |.72] .74] 62.4 60.6| 10006] 
109 57.14 1.70] .99 -O| .93 -.4] |.81] .74] 47.7 60.3] 10007] 
104 «49.83 1.76]1.24 1.7|1.10 -6] |.68| .71] 64.4 64.1| 10002] 
109 44.35 1.77]|1.00 +1] .91 -.4] |.67] .70] 56.9 67.2] 10003] 
109 48.48 1.82]1.03 -2| .86 -5| |.66| .68] 69.7 67.5| 1¢ee4| 
109 = 335.60 1.92] .88 -.8| .92 -.2] |.68|] .65] 67.0 68.5| 10008] 
109 32.54 2.00] .52 -3.9| .42 -2.1] |.75] .62] 91.7 71.1] 10009] 
errrrrrr rrr errr rere Peers Seer rerrs Sir er Peer errr Serer | 
108.4 50.08 1.79] .98 -4| .93 -.4] | 64.4 65.1] l 
16: 22.18 -10] .38 «=62.7| .41 2.4] | 13.9 3.7] | 


| $.0. 39.5 
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Confirmation of Dimensionality 


Determining the unidimensionality of an instrument is crucial to ensure accurate results 
are obtained from the study. The Principal Component Analysis of Rasch Residual (PCAR) 
is frequently used to detect unidimensionality. Based on the results presented in Figure 5, the 
Raw Variance Explained by Measures value's observed value is 61.1%, which according to 
Sumintono and Widhiarso (2015), is an excellent level of unidimensionality. This indicates that 
the ATT instrument is capable of measuring algebraic thinking skills. The unexplained variance 
in the first contrast was found to be 9.5%, which is below the threshold of 15%. Therefore, it 
can be concluded that the ATT instrument has strong unidimensionality. 


Figure 5 
Principal Component Analysis of Rasch Residual (PCAR) 


INPUT: 109 Person 9 Item REPORTED: 109 Person 9 Item 4 CATS | WINSTEPS 3.73 


Table of STANDARDIZED RESIDUAL variance (in Eigenvalue units) 
==tmpirical -=— Modeled 


Total raw variance in observations - 23.1 100.0% 100.8% 
Raw variance explained by measures = 14.1/| 61.1% 60.8% 
Raw variance explained by persons = 7.7 33.5% 33.3% 
Raw Variance explained by items - 6.4 27.6% 27.5% 
Raw unexplained variance (total) - 9.0 38.9% 100.0% 39.2% 
Unexplned variance in ist contrast = 2.2 9.5% 24.5% 
Unexplned variance in 2nd contrast = 1.8 7.8% 20.2% 
Unexplned variance in 3rd contrast 1.5 6.6% 17.0% 
Unexplned variance in 4th contrast = 1.2 5.3% 13.7% 
Unexplned variance in 5th contrast = -9 4.0% 10.2% 


Reliability and Separation Indices 


To assess ATT's reliability, person measure reliability was used to distinguish students 
based on their responses, while item measure reliability was used to differentiate between 
items based on their approved authenticity by respondents. Table 9 presents an overview of 
the reliability findings. Based on Cronbach's alpha coefficient (KR-20), Table 9 indicates that 
ATT has strong reliability, with a value of .90. The item reliability score of .98 is classified 
as excellent (Sumintono & Widhiarso, 2015), and the item separation index of 6.29 is greater 
than 3.0, suggesting that ATT has a good distribution of items. According to Krishnan and 
Idris (2014), for instruments to accurately measure students, the separation of respondents 
should surpass 1.00. An item separation value above 2.00 is considered good (Linacre, 2003). 
Ardiyanti (2016) proposed an equation, H = [(4 x separation index) + 1] / 3, to determine the 
number of strata of separate items (H) based on the separation index. The resulting H value in 
this study is 8.72, rounded up to 9, indicating that ATT has nine levels of items. 
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Table 9 


Summary of the Reliability Findings 


Statistics Value Interpretation 
Cronbach Alpha Rated (KR -20) 90 High 

Item Reliability 98 Excellent 
Item Separation Index 6.29 Good 
Person Reliability 86 Good 
Person Separation Index 2.45 Good 


PROBLEMS 
OF EDUCATION 


IN THE 21* CENTURY 


Vol. 81, No. 2, 2023 


237 


In addition, Rasch's analysis also recorded a good respondent reliability value of .86 
(Sumintono & Widhiarso, 2015). Bond and Fox (2007) also supported that the respondent's 
reliability rating higher than .80 indicates that respondents provide good and consistent 
feedback. The separation value of 2.45 is 'good' if its value exceeds 2.00 (Linacre, 2003). The 
separation value of 2.45 produces strata, H = 3.60 (can be rounded to 4). This value shows that 
students can be divided into four groups (weak, moderate, good and excellent) according to 
their responses based on items in ATT. 


Figure 6 

Principal Component Analysis of Rasch Residual (PCAR) 
INPUT: 109 Person 9 Item REPORTED: 109 Person 9 Item 4 CATS  WINSTEPS 3.73 

SUMMARY OF 109 MEASURED Person 

| TOTAL MODEL INFIT ouIFIT =| 
| SCORE COUNT MEASURE ERROR MNSQ 2STD MNSQ STO | 
| -nnnneeneenneenenenneeneeeenennneennnenne ene tenneneeeeneenneeeseenseeteee es | 
| MEAN %.3 9.8 69.39 6.56 -96 8 .93 1 | 
| $.0. 5.2 a 18.99 1.53 37 8 46 8 | 
| max, 44.0 9.8 103.99 10.97 1.62 1.3 2.10 1.3 | 
| MIN. 25.8 8.8 27.47 5.58 5 1.7 .21 .2 9 
Joccsncanccccesacescenccscecccnssccsssanqsscescsusscsessescessscsessesensscons | 
| REAL RMSE 7.14 TRUE SO 17.50 [SEPARATION 2.45 Person RELIABILITY .86 || 
|MODEL RMSE 6.74 TRUE SD 17.66 erson 87 | 
| S.E. OF Person MEAN = 1.82 | 


Person RAW SCORE-TO-MEASURE CORRELATION = .98 
CRONBACH ALPHA (KR-28) Person RAW SCORE “TEST” RELIABILITY = .98 


SUMMARY OF 9 MEASURED Itee 


| TOTAL MODEL INFIT OUTFIT =| 
| SCORE COUNT MEASURE ERROR MNSQ ZSTD MNSQ ZSTO | 
|-------nnnnnnnnnnnnnnerenennsnennenerennenennenteneeserenennenentennrenneness | 
| MEAM =. 439.1 108.4 50.08 1.79 6S. =4 <8: =f) 
| $.0. 39.5 1.6 12.11 -10 a ee ee 
| max, 495.0 109.0 71.41 2.00 1.85 4.9 1.94 5.1] 
| MIN 378.8 104.0 32.54 1.78 ae es ee 
|oocenccncncencnncnenncnenceeecernnnsnssssnsnssnsnssssnssssnsssssssssscnsesses | 
| REAL RMSE 1.90 TRUE SO 11.96 SEPARATION 6.29 Item RELIABILITY .98|| 
|MODEL RMSE 1.88 TRUE SD 11.97 SEPARATION 6.66 Item RELIABILITY .98 | 


| S.E. OF Item MEAN = 4.28 
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Discussion 


The research conducted a comprehensive analysis of the validity and reliability of the 
ATT instrument for measuring algebraic thinking skills among seventh graders from secondary 
school, as summarized in Table 5 to Table 9. The validity analysis covered item suitability, 
polarity, and dimensionality. 

After conducting an item conformity analysis, it was found that all items in the ATT 
instrument were deemed suitable for use as they met the criteria of Outfit MNSQ, Outfit ZSTD 
and PTMEA-CORR. A positive PTMEA-CORR analysis indicates that all instrument items 
are consistent in interpreting the measured constructs (Bond & Fox, 2007). Moreover, the Raw 
Variance Explained by Measures of the ATT instrument analysis shows a value of over 60%, 
indicating good dimensionality and confirming the instrument's ability to measure algebraic 
thinking skills in three constructs: generalized arithmetic, function, and modelling. 

The reliability assessment of the ATT instrument indicates a high level of trustworthiness 
in measuring algebraic thinking skills among seventh graders. This is supported by the very 
high Cronbach's alpha value, the excellent reliability value of the instrument items, and the good 
reliability of the respondents. The ATT instrument also shows the good item and respondent 
separation values, indicating that the instrument has nine strata according to the number of 
items in ATT and the respondents are divided into four strata based on their level of ability 
(weak, medium, good, excellent). 

While previous research has produced several instruments for assessing elementary 
students’ level of algebraic thinking skills, such as those developed by Ralston et al. (2018), 
Alghtani et al. (2010), and Somasundram et al. (2016), there has been a need for a tool to assess 
these skills among secondary school students. ATT was developed to address this limitation 
and is explicitly designed to evaluate the level of algebraic thinking skills in problem-solving. 
Beyond its use in determining the extent of students' algebraic thinking skills, ATT also offers 
an important means of analyzing these skills in secondary Mathematics Education. 

In general, the validity and reliability of ATT were evaluated using the Rasch Measure- 
ment Model. The ATT instrument focused on three constructs: Generalized Arithmetic, Func- 
tions, and Modelling. To ensure content validity, four experts reviewed all nine open-ended 
items. Item two was revised to avoid confusion in language writing. Additionally, the question's 
keyword was changed from "original perimeter of a garden" to "original perimeter of a rect- 
angular shape garden" to encourage high-level thinking among students. The third item was 
modified to ensure that the meaning of sentences in Malay and English is consistent. Experts 
suggested that the question's intent should be aligned in both languages. 

The research suggests that ATT, a new instrument in secondary mathematics education, 
aligns with Kaput's algebraic thinking skills model and can provide educators with valuable data 
on the extent of students' algebraic thinking skills. This data can be used to develop appropriate 
educational programs aimed at enhancing students' algebraic thinking abilities. Earlier studies, 
such as Hiebert et al. (1999), have established the significance of possessing algebraic thinking 
skills and have shown that it is a robust indicator of success in mathematics in the future. The 
findings of this study suggest that an appropriate algebraic thinking model can be used to assess 
the level of algebraic thinking skills among secondary students. Moreover, the ATT instrument 
utilizes problem-solving questions to enable learners to effectively tackle complex and non- 
routine mathematical problems. These results further support the idea that a strong foundation 
in algebraic thinking is critical for students to develop problem-solving skills and excel in 
mathematics, as emphasized by Fong and Lee (2009). 

Furthermore, it is worth noting that while most previous instruments developed to assess 
algebraic thinking have used the exploratory factor analysis method, the ATT instrument employs 
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Rasch analysis. This approach is associated with greater accuracy in instrument development, 
which is an important advantage of ATT over previous instruments. Using Rasch analysis, the 
ATT instrument is better equipped to accurately measure students’ algebraic thinking skills and 
provide valuable insights into areas where students may need additional support to improve 
their mathematical abilities. 


Conclusion and Implications 


Constructing an algebraic thinking skills instrument with a high level of validity and 
reliability is essential to ensure that the developed instrument can be used repeatedly. This 
is particularly important in enabling researchers to make informed decisions based on the 
instrument's findings. The results of this study also support the use of Rasch analysis in the 
development of the ATT instrument for measuring the algebraic thinking skills of students in 
other areas. By demonstrating the validity and reliability of the instrument using the Rasch 
Measurement Model, this study offers a strong foundation for its use in measuring the algebraic 
thinking skills of seventh-grade students in secondary schools. 

This study has important practical and methodological implications. From a practical 
standpoint, the Algebraic Thinking Test (ATT) is a new instrument developed based on the 
algebraic thinking model. The algebraic thinking model can be adapted to suit the context of 
mathematics education in any country. The ATT instrument is well-suited to address the gap in 
mathematics education research, specifically in measuring algebraic thinking in problem-solv- 
ing among secondary school students. The development of this instrument is a significant con- 
tribution towards ensuring the continuity of the application of algebraic thinking in mathemat- 
ics education. 

On the methodological side, the Rasch Measurement Model used to assess the validity 
and reliability of the ATT instrument is highly specific and detailed. This model's analysis indi- 
cates that the ATT instrument is highly valid for measuring algebraic thinking in problem-solv- 
ing based on item-person fit, item polarity, and unidimensionality. High-reliability analysis of 
the ATT instrument, as indicated by a good item-person separation value, proves its reliability 
in measuring algebraic thinking involving three constructs: Generalized Arithmetic, Functions, 
and Modelling for seventh-grade students. Validity and reliability analysis of the ATT instru- 
ment is essential for accurate measurement of the variables studied and informed decision-mak- 
ing based on analysis of the findings. Furthermore, these findings provide a basis for other 
researchers to use the ATT instrument in assessing algebraic thinking in mathematics education 
among seventh-grade students in other contexts. Ultimately, the Rasch Measurement Model has 
successfully demonstrated the ATT instrument's suitability for actual field study in assessing al- 
gebraic thinking in mathematics education among seventh-grade students in secondary schools. 

While the results demonstrate the reliability and validity of the ATT instrument for as- 
sessing algebraic thinking skills in secondary school, it is important to acknowledge its limita- 
tions. The study was conducted in rural schools and involved a sample size of only 120 students, 
which may not represent the broader population of secondary school students. Future research 
should expand the sample size to include urban schools to increase generalizability. Moreover, 
ATT can be integrated into STEM curriculum models to enhance critical and inventive thinking 
skills in Mathematics. To enhance the generalizability of the findings, it is recommended to 
extend the research of ATT to other regions and diverse student populations. 
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